Skip to content

cluster: decode_deterministic 三值 enum + honest gate 抽 helper + CLI 值域 (#118, #120) - #130

Merged
kiki830621 merged 2 commits into
mainfrom
idd/cluster-determinism-enum
Aug 1, 2026
Merged

cluster: decode_deterministic 三值 enum + honest gate 抽 helper + CLI 值域 (#118, #120)#130
kiki830621 merged 2 commits into
mainfrom
idd/cluster-determinism-enum

Conversation

@kiki830621

@kiki830621 kiki830621 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Refs #118
Refs #120

Summary

Cluster PR — #120 item 1 and #118 are the same code: the extracted helper's return type is the enum #118 settled on. Doing them apart would have written the helper twice.

#118 — decode_deterministic was Bool?, and nil meant two different things: "legacy row" and "this backend ignores the flag". The new DecodeDeterminism enum records what we know, not what the backend is:

value meaning
deterministic-enforced the backend consumed --decode-deterministic, and it was on
fallback-enabled consumed, off (temperature fallback live)
flag-not-consumed the backend ignores the flag — and this makes no claim about whether its decode is actually reproducible

Legacy rows stay on field-absence, which is now unambiguous. flag-not-consumed deliberately does not say "this backend is deterministic": we know the flag wasn't consumed, we don't know what happens inside Parakeet / mlx-audio / Apple.

#120 item 1 — the honest gate was inline in the benchmark loop, so the "never lie" invariant could not be tested without running a real backend. It's now DecodeDeterminism.forBackend(_:flagRequested:), a pure function beside flagConsumingBackends, unit-tested across whisperkit / whisper.cpp / mlx-audio / all three Fluid backends / an unknown future backend.

#120 item 2 — --run-kind is now a RunKind: ExpressibleByArgument, so a typo dies at the CLI boundary (exit 64) instead of reaching bench CI.

#120 item 3 landed on the bench side (PsychQuant/bestASR-bench@c93a70e): both provenance fields now accept explicit null as equivalent to absent, and the value sets are hoisted to module constants.

Timing

The migration was free only right now: bench had 156 rows, zero carrying either field, and no sweep had run since #111 merged. The next release sweep starts accumulating rows in whatever representation is current — this deliberately-breaking change (old true/false are now rejected) had to land before that.

Verification

Independently re-verified by the orchestrator, not taken on the implementer's report:

  • swift build clean; swift test 396 tests / 83 suites, 0 failures (baseline on this branch is 389 — the implementer corrected my brief, which had carried 402 over from the Bundle A branch that adds 13 tests of its own).
  • Cross-repo value sets match exactly — extracted from the Swift enums (including implicit rawValues where the case name is the wire value) and diffed against bench's RUN_KINDS / DECODE_DETERMINISM. This is the contract neither side can check alone.
  • CLI rejection is real: --run-kind bogus → exit=64, Error: The value 'bogus' is invalid … Please provide one of 'release-sweep' and 'adhoc'; both legal values pass parse.
  • bench TDD: run_kind: null was rejected before / accepted after; the enum values were rejected before / accepted after; old true/false now rejected; 156 existing rows still OK.

Checklist

Generated by /idd-all cluster mode. Do NOT add a GitHub close trailer — IDD discipline requires manual /idd-close after merge.

@kiki830621
feat: decode_deterministic becomes a three-value enum + extract the h…
7952ecd 
…onest gate (#118, #120)

#118 settled the representation: the field records WHAT WE KNOW, not what the
backend is. Bool? overloaded nil to mean both 'legacy row' and 'this backend
ignores the flag'. DecodeDeterminism separates them —

  deterministic-enforced  flag consumed, on
  fallback-enabled        flag consumed, off
  flag-not-consumed       backend ignores the flag, and this makes NO claim
                          about whether its decode is actually reproducible

Legacy rows stay on field-absence, which is no longer overloaded. An unknown
wire value throws DecodingError.dataCorrupted (loud), while absent and explicit
null both decode to nil.

#120 item 1: the honest gate moves out of the benchmark loop into
DecodeDeterminism.forBackend(_:flagRequested:), a pure non-optional function
next to flagConsumingBackends. It lives on the enum because the rule IS the
meaning of the cases — adding a flag-consuming backend is now one edit in the
file that documents the invariant it can violate. The 'never lie' property is
finally unit-testable without running a real backend.

#120 item 2: --run-kind is a RunKind ExpressibleByArgument (mirroring the
existing HallucinationFilterMode precedent), so a typo fails at the CLI boundary
with exit 64 instead of surviving to bench CI. MeasurementRow.runKind stays
String? — the type constrains the entry point, not the stored row.

Both value sets carry a keep-in-sync note pointing at bench's
validate_measurements.py; there is no mechanical link (the #120 Residue).
bestASR-bench side landed as c93a70e.

Refs #118 Refs #120
@kiki830621 kiki830621 mentioned this pull request Aug 1, 2026
Open
@kiki830621
fix: verify round-1 — honest docs, correct citation, lock the old-boo…
a9692d3 
…l shape (#130)

Seven convergent verify findings, all in-scope by the DA's boundary (this PR may
fix text it wrote and text it made false, plus tests in the file it created —
not what it merely made visible):

- CHANGELOG cited bestASR-bench@e728f1a as the authority for the enum. That
  commit's validator enforces isinstance(..., bool) — it REJECTS every value the
  paragraph introduces. Now c93a70e. Two legs caught it independently; shipped
  text that contradicts itself is the worst kind.

- The compat sentence covered only an ABSENT field. It now also states what
  happens to the boolean this same unreleased entry used to describe: rejected,
  not coerced, with the evidence that nothing ever wrote one.

- #120 was missing from the CHANGELOG entirely, though --run-kind bogus failing
  at parse is user-visible. Added, including WHY the stored field stays a string.

- The #109 paragraph said --decode-deterministic is 'forwarded elsewhere',
  which now reads as contradicting the #118 paragraph beside it. It is a silent
  no-op elsewhere — that is the whole reason flag-not-consumed exists.

- deterministicEnforced's doc claimed 'the same audio yields the same text'. An
  over-claim inside a PR about not over-claiming — disabling temperature
  fallback does not establish byte-identical output across runtime, hardware or
  model revision. It now says what was observed: the setting was enforced.

- 'hyphenated snake-case' is kebab-case.

- RunKind's doc claimed to be the Swift side's single source of truth. It is
  not: the row types stay String?, so a library caller can still store anything
  and bench CI stays the backstop. The comment now states that scope AND the
  reason it differs from decode_deterministic — that field's domain is derived
  from this repo's backend roster, while run_kind is human-typed provenance
  whose vocabulary is already incomplete (the regression gate benchmarks with no
  --run-kind at all). Closing it at the row type would trade a loud CI failure
  for a silently dropped row.

Plus two tests: the pre-#118 boolean shape is now locked as REJECTED (a census
expires, a test does not — and main still contains the bool-writer today), and
SubmissionRow's absent-not-null contract is asserted on the type that actually
crosses to the bench repo, not only via MeasurementRow.

Forward-compat is now documented on the enum: adding a case is breaking for
older readers AND fails invisibly, because both consumers turn the throw into a
quiet drop. Mechanism-level fix tracked in #131.

Refs #118 Refs #120
kiki830621 added a commit to PsychQuant/bestASR-bench that referenced this pull request Aug 1, 2026
@kiki830621
docs: name the actual non-consuming backends, not a hypothetical one
092e142 
The flag-not-consumed row listed 'Apple', which is not a backend in ModelGrid —
Apple Speech is still an open proposal (bestASR#121). Name what exists:
mlx-audio (silent no-op) and the Fluid family (no such knob).

Refs PsychQuant/bestASR#130
@kiki830621

kiki830621 commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Verify Report — PR #130 (#118, #120)

Engine

manual fan-out — 3 lens Agents（logic / requirements / regression，model: opus）

  • sequenced Devil's Advocate（opus）+ Codex（gpt-5.6-sol xhigh）跨模型盲驗。5 legs。

Aggregate

PASS — 0 blocking；7 個 in-scope fix 已於 round-1（a9692d3 + bench 092e142）修畢；
1 個 follow-up 已 file（#131）。DA 明確表示無法推翻「safe to land」，且反而把安全論證做得更強。

要求覆蓋率

8 / 8 FULLY（#118 契約 5 項 + #120 三項；requirements lens 逐項對照 Decision comment）。

這個改動是「刻意破壞相容」——安全性用六種方法查證

舊 decode_deterministic: true/false 現在被拒。正當性建立在「野生資料 0 筆」，而該事實被
獨立查證六次（三個 leg 各自 + DA 追加三條）：

# 方法 結果 來源
1 bench 156 rows 掃描 零攜帶 regression
2 本機 store 364 rows 的解析後 key 聯集（非 grep） 兩欄位皆不存在 regression
3 日期相關性：#111 於 07-31 06:27 merge，最新本機 row 為 07-30 沒有任何測量是用能寫出該欄位的 binary 產生的 regression
4 bench git object store 每一個 blob（--batch-all-objects，涵蓋已刪除/amend/orphaned） 唯一 bool 命中是 SUBMISSION_FORMAT.md 的文件範例 DA
5 bench 零 fork、單一人類貢獻者、單一 merged PR 無其他發布者 DA
6 ssh mini / ssh kyle 皆無 ~/.bestasr、無 checkout；machines.jsonl 兩個 machine_id 是同一台 M5 Max 從未部署到本機以外 DA

DA 另加一條：.build/release/bestasr 日期為 07-30 12:40（早於 #111 merge），且 bestasr
不在 PATH 上——能寫出 bool 的 binary 從未存在過。

Findings（合併後，Fix 欄 = round-1）

# Severity Finding Source Action
1 PASS Codable 三態正確：absent→nil、explicit null→nil、未知字串→throw。logic lens 用真實的 .sortedKeys/.iso8601 encoder 與 store encoder 驗證，非預設 JSONEncoder agents:logic+codex ✅
2 PASS nil 編碼為省略 key 而非 null（跨 repo load-bearing 契約），在 MeasurementRow 與 SubmissionRow 皆成立 agents:logic+codex ✅
3 PASS 跨 repo 值域逐字相同。requirements 用 implicit-rawValue-aware parser 提取（case adhoc 無顯式 rawValue 是真陷阱——orchestrator 的 naive grep 曾誤報不一致），並把三個值端到端推過真實 validator agents:requirements ✅
4 PASS flagConsumingBackends = 恰好那兩個真的讀 options.deterministicDecode 的 engine；externalCapable = [.mlxAudio] 阻止 adapter 偽裝成 whisper 家族 agents:logic+regression+codex ✅
5 PASS fallback-enabled 語意經實測確認：WhisperKit 預設 temperatureFallbackCount 量到 5，whisper.cpp 不帶 -nf——不帶 flag 確實代表 fallback 仍作用 agents:logic ✅
6 PASS 測試非空洞：requirements 做 5 個 mutation 全 RED，其中 adhoc→"ad-hoc" 的 wire drift 同時在兩處變紅 agents:requirements ✅
7 PASS 無 scope creep（8 檔）；baseline 389/83 → 396/83，+7 完全對得上（7 個 @Test 新增、0 移除，既有測試就地修改而非刪除） agents:regression ✅
8 HIGH CHANGELOG 引用 e728f1a 當 enum 的 bench 依據，但該 commit 的 validator 強制 isinstance(..., bool)——會拒絕該段落介紹的每一個值。已發布文字自相矛盾 agents:requirements+regression（2 legs） ✅ Fixed a9692d3 → c93a70e
9 HIGH 舊 bool 值拋的是 typeMismatch(String) 而非實作者宣稱的 dataCorrupted（後者是給未知字串的），且 BenchmarkStore.load() 會丟掉整筆 row。此行為無測試鎖住，而 CHANGELOG 的相容性句子只涵蓋 absent agents:logic(F1) ✅ Fixed a9692d3（加測試鎖定「拒絕而非遷移」+ CHANGELOG 補述）
10 MEDIUM RunKind 只是邊界裝飾，不是不變式。機械證明：DecodeDeterminism 出現在 4 檔，RunKind 只在 DataModels + CLI；未驗證路徑每一跳都是 public，MeasurementRow(runKind: "relase-sweep") 仍可編譯——正是該註解宣稱已修好的失效模式。同一 commit 對兩個姊妹欄位做了相反的 forward-compat 選擇且未陳述理由 agents:logic(F7)+regression(F2)+codex（3 legs） ✅ Fixed a9692d3（DA 裁決保留 String?，把註解收斂為 CLI-scoped 並寫出理由——見下方裁決）
11 MEDIUM enum 開放域的 forward-compat：未來加第 4 個值對舊 client 是 breaking，且靜默失敗 agents:logic(F2)+codex ✅ 文件層 Fixed a9692d3；機制層 → #131
12 HIGH BenchmarkStore.Snapshot.warnings 有零個非測試讀取者。grep -rc warnings Sources/bestasr/ 全 CLI target 零命中；每個 load() 呼叫端都丟棄它。唯一讀者是 4 個測試斷言（註解寫 // still loud）。所謂「大聲降級」只存在於單元測試裡——使用者兩條路徑都什麼也看不到 devils-advocate（五個 leg 中唯一發現） Follow-up → #131
13 MEDIUM 掉一筆 row 不會留下空缺：latestMeasurements 是 append-only 上的 latest-wins，231 個 key 中 30 個有 supersession 歷史（一個有 18 筆）→ 掉最新那筆會靜默把已被取代的舊值升為現值。對「新值寫在最新 row 上」的變更是最壞形狀 devils-advocate Follow-up → #131
14 LOW deterministicEnforced 的註解宣稱「same audio yields the same text」——在一個以「不過度宣稱」為立論的 PR 裡過度宣稱。停用 temperature fallback 不足以保證跨 runtime / 硬體 / 模型 revision 的位元一致 codex+agents:logic（2 legs） ✅ Fixed a9692d3
15 LOW #109 的 CHANGELOG 段落寫 --decode-deterministic「forwarded elsewhere」，與旁邊的 #118 段落矛盾（那正是 flag-not-consumed 存在的理由） agents:logic ✅ Fixed a9692d3
16 LOW 「hyphenated snake-case」術語錯誤——是 kebab-case codex ✅ Fixed a9692d3
17 LOW bench SUBMISSION_FORMAT.md 的 flag-not-consumed 列出「Apple」backend，但 ModelGrid 沒有它（#121 仍是提案）——orchestrator 自己的編輯錯誤 agents:regression(F5) ✅ Fixed bench 092e142
18 INFO #expect(throws: (any Error).self) 任何 throw 都滿足，斷言偏弱 agents:logic Noted（DA 判為 follow-up 邊界外，未修）
19 INFO logic F2 的「duplicate submission」被高估——dedupeKey 是 machine-scoped，只有自己機器已發布的 row 有關；需要降級情境，非預設路徑 devils-advocate（反向修正） Noted

DA 的兩項裁決（採納）

Q2 — RunKind 不對稱：裁定 (b) 保留 String? + 註解收斂為 CLI-scoped，理由比
orchestrator 原本的更強：decode_deterministic 的值域衍生自本 repo 自己的 backend roster
（對其為 total function），而 run_kind 是人工輸入的 provenance，其詞彙已可證明不完整——
scripts/regression-gate.sh 跑 benchmark 時根本不帶 --run-kind。把它在 row 型別上封閉，
等於用一個大聲的 CI 失敗換一次靜默的 stale-data 讀取。
DA 另指出一個倒置：KEEP-IN-SYNC 警告掛在寬鬆的欄位上，嚴格性掛在漂移時會靜默的那個欄位上。

Q5 — 三個值的命名（使用者拍板）：裁定保留，且理由優於 orchestrator 的：
deterministic-enforced 是機制無關的——fallback-disabled 會在某天某個 backend 改用
seeding 而非停用 fallback 來達成確定性時失效；而且它追蹤的是 flag 自己的名字。
謊言在那句註解，不在名字。 故不需回頭請使用者重新裁決。

Scope Check

無超出 #118/#120 範圍。Round-1 的 7 項修復嚴格遵守 DA 的邊界判準：
「這個 PR 可以修它自己寫的文字、它自己弄假的文字、以及它自己建立的檔案裡的測試——
不修它只是讓人看見的東西。」

Process Gaps

無。Codex leg 正常完成。

Follow-up Findings Filed

誠實的範圍聲明

本 PR 交付的是 decode_deterministic 的三值表示法 + honest gate 的可測性 + --run-kind 的
CLI 值域。不宣稱：(1) run_kind 在型別層封閉（刻意不封，理由已寫進 code）；
(2) 未來擴充 enum 是安全的（是 breaking 且靜默，見 #131）；(3) deterministic-enforced 保證
輸出可重現（只保證該設定被強制執行）。

This was referenced Aug 1, 2026
Closed
Closed
Merged
@kiki830621
kiki830621 merged commit a4ee7c5 into main Aug 1, 2026
1 check passed
@kiki830621
kiki830621 deleted the idd/cluster-determinism-enum branch August 1, 2026 15:59
kiki830621 added a commit that referenced this pull request Aug 1, 2026
@kiki830621
merge main into #121 branch
da7f56f 
Git auto-merged cleanly but silently mis-filed the CHANGELOG: it produced
a duplicate '### Fixed' heading and moved the #109 (release sweep) and
#111 (provenance fields) entries — both Added items — underneath it.
Restructured so Unreleased has one Added (#121, #109, #111), one Fixed
(#121 x6, #117, #116/#115, #112) and one Changed (#107).

Verified against the post-merge main: 445 tests / 88 suites pass, which
covers the semantic interaction CI could not see (#130 changed
decodeDeterministic's type after this branch's CI last ran).

Refs #121
kiki830621 added a commit that referenced this pull request Aug 1, 2026
@kiki830621
feat: Apple Speech (SpeechAnalyzer) backend (#121) (#135)
4ac8b61 
* feat: Apple Speech (SpeechAnalyzer) backend (#121)

The OS-native ASR backend: Speech.framework's SpeechAnalyzer /
SpeechTranscriber, macOS 26+. No dependency, no weight download, no
supply-chain pin — the model ships with the OS, so its version is the
OS version.

Registered unconditionally in CommandCore.live(); the struct carries no
@available so it is constructible at the package's macOS 14 deployment
target, and every macOS-26-only call sits behind an @available helper.
isAvailable() is a pure version gate and never touches the asset
inventory, which keeps list-backends cheap.

Three probed behaviors drive the design:

- transcriber.results is an AsyncSequence that must be consumed before
  audio is fed, or results are lost.
- A missing locale asset reports "Audio format is not supported" — a
  misleading message. The engine checks installedLocales and downloads
  on demand instead of diagnosing from the error text.
- The framework has no determinism knob, so decode_deterministic
  records flag-not-consumed rather than claiming enforcement (#118).

Language is required rather than guessed: SpeechTranscriber has no
auto-detect mode, and a ja file decoded under en_US yields garbage that
would enter the benchmark evidence base silently. zh maps to zh_TW to
match the project's Common Voice zh-TW corpora. Per-segment confidence
is derived as the minimum over Apple's per-run values, so a confident
run cannot mask a weak one inside the same cue.

Also fixes a defect this backend exposed: Router's backend-membership
list omitted apple-speech, so an explicit --backend was discarded and
another backend's output was written under the user's chosen name,
behind a false "unavailable" warning that --explain alone revealed.

Refs #121

* fix: verify-round corrections to the Apple Speech backend (#121)

Six defects found by the verification legs, four of them capable of
producing plausible-but-wrong output rather than an error.

Locale resolution read subtags BY POSITION, so the script subtag in
zh-Hans was taken for a region, matched nothing, and fell through to
the zh preference: a user who asked for Simplified silently received
the Traditional model (confirmed by probe). Subtags are now classified
by shape, with Hans/Hant mapped to CN/TW and an explicit region still
outranking a contradicting script.

Provenance recorded the REQUESTED language rather than the one that
ran, so a pt-BR request served by pt_PT was labelled pt-BR. The
resolved locale is now recorded — hyphenated, because Apple's
underscored zh_TW defeats LanguageResolver.baseSubtag (which splits on
"-" only) and would have silently disabled the D7 zh script fold for
this backend.

Recognized text with unusable timing was dropped. The WER denominator
is the reference word count and does not move, so a dropped correct
word adds a deletion while a dropped garbage word removes an insertion
— the measured rate moves in an unsignposted direction. That now fails
the run. Whitespace-only segments are kept, since the seam joins with
no separator.

The locale asset was installed before the audio file was validated, so
a typo in the path could cost a multi-hundred-MB download. A nil
installation request also fell through to the misleading "Audio format
is not supported" this preflight exists to prevent; installedLocales is
now re-checked afterwards.

The unstructured collector Task was neither awaited nor cancelled when
the analyzer threw, pinning the transcriber across a benchmark sweep.

aggregateConfidence now treats NaN/inf/out-of-range as unknown rather
than passing them through as numbers.

Also corrects a claim that exceeded its evidence: "no weight download"
appeared in three places while this project itself had to download the
ja_JP asset. The accurate claim is about provenance (Apple rather than
a model hub), not about network traffic.

Refs #121

* docs: declare the Xcode 26 / SDK 26 build floor raised by apple-speech (#121)

The package still RUNS on macOS 14, but since #121 it no longer BUILDS on
an older toolchain: @available gates the runtime, not the type checker, so
Xcode 16 fails on SpeechAnalyzer / SpeechTranscriber. CI is green only
because the workflow selects the newest Xcode on the runner (Swift 6.2.4 =
Xcode 26); nothing declared the requirement.

Refs #121

* merge main into #121 branch

Git auto-merged cleanly but silently mis-filed the CHANGELOG: it produced
a duplicate '### Fixed' heading and moved the #109 (release sweep) and
#111 (provenance fields) entries — both Added items — underneath it.
Restructured so Unreleased has one Added (#121, #109, #111), one Fixed
(#121 x6, #117, #116/#115, #112) and one Changed (#107).

Verified against the post-merge main: 445 tests / 88 suites pass, which
covers the semantic interaction CI could not see (#130 changed
decodeDeterministic's type after this branch's CI last ran).

Refs #121
@kiki830621 kiki830621 mentioned this pull request Aug 1, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kiki830621